{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "punL79CN7Ox6"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "_ckMIh7O7s6D"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ph5eir3Pf-3z"
      },
      "source": [
        "# Constructing a Text Generation Model\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "S5Uhzt6vVIB2"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c03_nlp_constructing_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c03_nlp_constructing_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7GbGfr_oLCat"
      },
      "source": [
        "Using most of the techniques you've already learned, it's now possible to generate new text by predicting the next word that follows a given seed word. To practice this method, we'll use the [Kaggle Song Lyrics Dataset](https://www.kaggle.com/mousehead/songlyrics)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4aHK2CYygXom"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2LmLTREBf5ng"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Other imports for processing data\n",
        "import string\n",
        "import numpy as np\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GmLTO_dpgge9"
      },
      "source": [
        "## Get the Dataset\n",
        "\n",
        "As noted above, we'll utilize the [Song Lyrics dataset](https://www.kaggle.com/mousehead/songlyrics) on Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4Bf5FVHfganK"
      },
      "outputs": [],
      "source": [
        "!wget --no-check-certificate \\\n",
        "    https://drive.google.com/uc?id=1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8 \\\n",
        "    -O /tmp/songdata.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gu1BTzMIS1oy"
      },
      "source": [
        "## **First 10 Songs**\n",
        "\n",
        "Let's first look at just 10 songs from the dataset, and see how things perform."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fmb9rGaAUDO-"
      },
      "source": [
        "### Preprocessing\n",
        "\n",
        "Let's perform some basic preprocessing to get rid of punctuation and make everything lowercase. We'll then split the lyrics up by line and tokenize the lyrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2AVAvyF_Vuh5"
      },
      "outputs": [],
      "source": [
        "def tokenize_corpus(corpus, num_words=-1):\n",
        "  # Fit a Tokenizer on the corpus\n",
        "  if num_words > -1:\n",
        "    tokenizer = Tokenizer(num_words=num_words)\n",
        "  else:\n",
        "    tokenizer = Tokenizer()\n",
        "  tokenizer.fit_on_texts(corpus)\n",
        "  return tokenizer\n",
        "\n",
        "def create_lyrics_corpus(dataset, field):\n",
        "  # Remove all other punctuation\n",
        "  dataset[field] = dataset[field].str.replace('[{}]'.format(string.punctuation), '')\n",
        "  # Make it lowercase\n",
        "  dataset[field] = dataset[field].str.lower()\n",
        "  # Make it one long string to split by line\n",
        "  lyrics = dataset[field].str.cat()\n",
        "  corpus = lyrics.split('\\n')\n",
        "  # Remove any trailing whitespace\n",
        "  for l in range(len(corpus)):\n",
        "    corpus[l] = corpus[l].rstrip()\n",
        "  # Remove any empty lines\n",
        "  corpus = [l for l in corpus if l != '']\n",
        "\n",
        "  return corpus"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "apcEXp7WhVBs"
      },
      "outputs": [],
      "source": [
        "# Read the dataset from csv - just first 10 songs for now\n",
        "dataset = pd.read_csv('/tmp/songdata.csv', dtype=str)[:10]\n",
        "# Create the corpus using the 'text' column containing lyrics\n",
        "corpus = create_lyrics_corpus(dataset, 'text')\n",
        "# Tokenize the corpus\n",
        "tokenizer = tokenize_corpus(corpus)\n",
        "\n",
        "total_words = len(tokenizer.word_index) + 1\n",
        "\n",
        "print(tokenizer.word_index)\n",
        "print(total_words)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v9x68iN_X6FK"
      },
      "source": [
        "### Create Sequences and Labels\n",
        "\n",
        "After preprocessing, we next need to create sequences and labels. Creating the sequences themselves is similar to before with `texts_to_sequences`, but also including the use of [N-Grams](https://towardsdatascience.com/introduction-to-language-models-n-gram-e323081503d9); creating the labels will now utilize those sequences as well as utilize one-hot encoding over all potential output words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "QmlTsUqfikVO"
      },
      "outputs": [],
      "source": [
        "sequences = []\n",
        "for line in corpus:\n",
        "\ttoken_list = tokenizer.texts_to_sequences([line])[0]\n",
        "\tfor i in range(1, len(token_list)):\n",
        "\t\tn_gram_sequence = token_list[:i+1]\n",
        "\t\tsequences.append(n_gram_sequence)\n",
        "\n",
        "# Pad sequences for equal input length \n",
        "max_sequence_len = max([len(seq) for seq in sequences])\n",
        "sequences = np.array(pad_sequences(sequences, maxlen=max_sequence_len, padding='pre'))\n",
        "\n",
        "# Split sequences between the \"input\" sequence and \"output\" predicted word\n",
        "input_sequences, labels = sequences[:,:-1], sequences[:,-1]\n",
        "# One-hot encode the labels\n",
        "one_hot_labels = tf.keras.utils.to_categorical(labels, num_classes=total_words)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Zsmu3aEId49i"
      },
      "outputs": [],
      "source": [
        "# Check out how some of our data is being stored\n",
        "# The Tokenizer has just a single index per word\n",
        "print(tokenizer.word_index['know'])\n",
        "print(tokenizer.word_index['feeling'])\n",
        "# Input sequences will have multiple indexes\n",
        "print(input_sequences[5])\n",
        "print(input_sequences[6])\n",
        "# And the one hot labels will be as long as the full spread of tokenized words\n",
        "print(one_hot_labels[5])\n",
        "print(one_hot_labels[6])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-1TAJMlmfO8r"
      },
      "source": [
        "### Train a Text Generation Model\n",
        "\n",
        "Building an RNN to train our text generation model will be very similar to the sentiment models you've built previously. The only real change necessary is to make sure to use Categorical instead of Binary Cross Entropy as the loss function - we could use Binary before since the sentiment was only 0 or 1, but now there are hundreds of categories.\n",
        "\n",
        "From there, we should also consider using *more* epochs than before, as text generation can take a little longer to converge than sentiment analysis, *and* we aren't working with all that much data yet. I'll set it at 200 epochs here since we're only use part of the dataset, and training will tail off quite a bit over that many epochs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "G1YXuxIqfygN"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\n",
        "model.add(Bidirectional(LSTM(20)))\n",
        "model.add(Dense(total_words, activation='softmax'))\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "history = model.fit(input_sequences, one_hot_labels, epochs=200, verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AXVFpoREhV6Y"
      },
      "source": [
        "### View the Training Graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "aeSNfS7uhch0"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.show()\n",
        "\n",
        "plot_graphs(history, 'accuracy')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1rAgRpxYhjpB"
      },
      "source": [
        "### Generate new lyrics!\n",
        "\n",
        "It's finally time to generate some new lyrics from the trained model, and see what we get. To do so, we'll provide some \"seed text\", or an input sequence for the model to start with. We'll also decide just how long of an output sequence we want - this could essentially be infinite, as the input plus the previous output will be continuously fed in for a new output word (at least up to our max sequence length)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DC7zfcgviDTp"
      },
      "outputs": [],
      "source": [
        "seed_text = \"im feeling chills\"\n",
        "next_words = 100\n",
        "  \n",
        "for _ in range(next_words):\n",
        "\ttoken_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "\ttoken_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "\tpredicted = np.argmax(model.predict(token_list), axis=-1)\n",
        "\toutput_word = \"\"\n",
        "\tfor word, index in tokenizer.word_index.items():\n",
        "\t\tif index == predicted:\n",
        "\t\t\toutput_word = word\n",
        "\t\t\tbreak\n",
        "\tseed_text += \" \" + output_word\n",
        "print(seed_text)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "l10c03_nlp_constructing_text_generation_model.ipynb",
      "private_outputs": true,
      "provenance": [
        {
          "file_id": "1_08R0eJLNBvyfAnkQ6DW8hEPESpeqhSU",
          "timestamp": 1586478923659
        }
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
